A researcher built a fictional billing database and had five AI models create semantic layers (business logic definitions) for it, then tested how well those layers helped the models answer finance questions. The results show that a well-built layer can dramatically improve accuracy, but models often make confident mistakes about scope and conventions that human finance teams would catch.

• Haiku failed to build a usable layer at all; the four models that succeeded got the math right but disagreed on scope—which accounts count as customers, whether subsidiaries roll up to parents, and how to handle QA test data.
• A reference layer written by a human improved Opus from 17 of 30 correct answers to 30 of 30, while a flawed layer built by Sonnet dropped Fable from 18 correct to 12, showing that semantic layers can either unlock or degrade model performance.
• Models built layers with high confidence regardless of accuracy: Muse matched finance definitions on all but one rule, while Opus and Fable made the same incorrect assumption about subsidiary rollups that finance never used, and Sonnet inconsistently filtered QA accounts across different metrics.

The article shows that when an LLM evaluates if text meets a given criterion, the answer already sits in its hidden state before any token is generated. By capturing the hidden representation at a designated seed token and training a small MLP head (with optional LoRA sharpening and isotonic calibration), you get a fast, calibrated classifier that accepts arbitrary English criteria without per-criterion retraining.

• LLMs encode the yes/no answer to a criterion in their hidden states before generating any text, so you can extract a calibrated probability from the hidden state at a seed token (~70% layer depth) instead of generating output.
• A small probe (MLP/linear head) trained on hidden states from a few thousand labeled triples, plus isotonic calibration, generalizes to arbitrary unseen criteria without retraining.
• Training a LoRA to produce one-sentence verdicts (but only reading the resulting hidden state, never generating) sharpens the signal by reshaping the residual stream geometry at the seed position.
• Prefilling content once and treating criteria as short continuations enables cache-efficient scoring against many criteria, though it can fail on cases requiring deep criterion-content interaction, which still need full cross-encoder prompts.

The article discusses the shifting landscape for data scientists and machine learning engineers in the age of large language models (LLMs). It emphasizes the importance of data science fundamentals in evaluating AI systems, addressing common pitfalls in metrics, experimental design, and data quality. The author argues that the core work of data scientists remains vital, even as their roles evolve.

• Off-the-shelf eval framework metrics often mislead teams; digging into your own data to find relevant metrics is what data scientists actually do.
• Using LLMs as judges without validating them against human labels is a common, risky shortcut.
• Synthetic test data that isn't grounded in real production logs leads to flawed experimental design and misleading results.
• Outsourcing labeling away from domain experts degrades data quality and undermines the whole evaluation process.